38. Quiz + Text: Recap

Recap

Wow! That was a ton. You learned:

1. How to set up hypothesis tests. You learned the null hypothesis is what we assume to be true before we collect any data, and the alternative is usually what we want to try and prove to be true.
   
   

2. You learned about Type I and Type II errors. You learned that Type I errors are the worst type of errors, and these are associated with choosing the alternative when the null hypothesis is actually true.
   
   

3. You learned that p-values are the probability of observing your data or something more extreme in favor of the alternative given the null hypothesis is true. You learned that using a confidence interval from the bootstrapping samples, you can essentially make the same decisions as in hypothesis testing (without all of the confusion of p-values).
   
   

4. You learned how to make decisions based on p-values. That is, if the p-value is less than your Type I error threshold, then you have evidence to reject the null and choose the alternative. Otherwise, you fail to reject the null hypothesis.
   
   

5. You learned that when sample sizes are really large, everything appears statistically significant (that is you end up rejecting essentially every null), but these results may not be practically significant.
   
   

6. You learned that when performing multiple hypothesis tests, your errors will compound. Therefore, using some sort of correction to maintain your true Type I error rate is important. A simple, but very conservative approach is to use what is known as a Bonferroni correction, which says you should just divide your \alpha level (or Type I error threshold) by the number of tests performed.

This lesson is often the most challenging for students throughout the entire nanodegree program.
In order to really have the ideas here stick, it can help to put them down in your own words.

Below are some quizzes to test that you are leaving with the main ideas from this lesson, as well as a link to a great blog post, written by one of your fellow classmates, to assist with the ideas of this lesson!

"What is a p-value?" blog post

QUIZ QUESTION: :

Review Quiz

Match the best related term to the definitions below regarding the lesson topics.

ANSWER CHOICES:

What we assume to be true before collecting any data.

The threshold for the percent of type I errors we are willing to commit.

The probability of observing our statistic or a more extreme statistic from the null hypothesis.

An alternative (and conceptually more simple for most) method of obtaining the results of a hypothesis test.

Definition	Term
alpha
p-value
confidence intervals from bootstrapping samples
null hypothesis

SOLUTION:

Definition	Term
alpha	The threshold for the percent of type I errors we are willing to commit.
p-value	The probability of observing our statistic or a more extreme statistic from the null hypothesis.
confidence intervals from bootstrapping samples	An alternative (and conceptually more simple for most) method of obtaining the results of a hypothesis test.
null hypothesis	What we assume to be true before collecting any data.

QUIZ QUESTION: :

Review Quiz

Match the best related term to the definitions below regarding the lesson topics.

ANSWER CHOICES:

These statistical techniques are generally aimed at helping us understand population parameters.

Techniques in this area are aimed at helping us draw conclusions about individuals in our population.

One method for correcting our type I error threshold when we perform more than one hypothesis test.

With larger sample sizes, this becomes less relevant.

Definition	Term
Machine Learning
Bonferroni Correction
Statistical Significance
Practical Significance
Hypothesis Testing & Confidence Intervals

SOLUTION:

Definition	Term
Machine Learning	Techniques in this area are aimed at helping us draw conclusions about individuals in our population.
Bonferroni Correction	One method for correcting our type I error threshold when we perform more than one hypothesis test.
Statistical Significance	With larger sample sizes, this becomes less relevant.
Hypothesis Testing & Confidence Intervals	These statistical techniques are generally aimed at helping us understand population parameters.